adapter: don't leak arrangement compression into the realized config - #38112
Draft
aljoscha wants to merge 6 commits into
Draft
adapter: don't leak arrangement compression into the realized config#38112aljoscha wants to merge 6 commits into
aljoscha wants to merge 6 commits into
Conversation
…duler The cluster controller has been default-on since v26.29 and owns the replica set of every user managed cluster. The break-glass dyncfg kept two legacy paths reachable: the REFRESH scheduler in `cluster_scheduling.rs`, whose two entry points already returned before doing any work while the gate was on, and the legacy branches in the ALTER sequencer. Delete the gate and everything only it kept alive: the scheduler module, its coordinator plumbing (two messages, the timer, the select-loop tick, the `cluster_scheduling_decisions` state), the two scheduler metrics, the `cluster_check_scheduling_policies_interval` system var, and the `ReplicaCreateDropReason::ClusterScheduling` variant. The persisted audit vocabulary stays: `SchedulingDecisionsWithReasonsV2` and friends are written by the controller's on-refresh path too, and old events must remain decodable.
`ManagedClusterIds` and the sequencer's two ownership tests excluded system/builtin clusters, so the sequencer kept a second, complete replica-materialization implementation alive just for them. The exception was originally load-bearing (the boot-time builtin replica migration and the controller would have been two conflicting writers), but `reconcile_builtin_cluster_replicas` now converges a builtin cluster's replica set on the cluster's own managed config, the same config the controller derives its targets from, so the two converge by construction. Drop the `is_user()` conjuncts. Runtime ALTERs of system clusters now flow through the controller like any other managed cluster: a config-shape change reshapes into a durable reconfiguration record, a factor change updates the config and the controller converges the replica set within a tick. The boot migration and the controller compose in both directions. The controller matches replicas by shape and count, never by name, so the migration-created `r1..rN` satisfy its baseline. The migration converges by canonical name, so a boot after a reshape renames or re-creates replicas the controller materialized under generator names, which is harmless churn on processes that are cold at boot anyway.
With the controller owning every managed cluster's replica set, `NeedsFinalization::Yes` has no producer. Delete the machine it drove: the `WaitForHydrated` and `Finalize` stages, the `-pending` overlap replicas, the `pending_cluster_alters` connection state and its retire paths, and the `AlterClusterWhilePendingReplicas` error. The direct reshape path is deliberately kept, as the synchronous cut-over. It is now routed to by an explicitly zero-timeout commit strategy (`WITH (WAIT FOR '0s')`, or `WAIT UNTIL READY (TIMEOUT '0s', ON TIMEOUT 'COMMIT')`) rather than by the absence of a `WAIT` clause. Two reasons. It is the escape hatch: every other reshape depends on the controller ticking and applying, and this is the one that still works when the controller itself is the problem, while also unsticking a wedged reconfiguration by retiring its record. And the semantics are honest: a zero timeout with commit already means "cut over now, hydrated or not", so doing it synchronously in the ALTER is the same outcome minus a tick. "The same outcome" has to be true, so the cut-over does not improvise. It folds its target onto an in-flight one exactly as the reshape path does, and it converges the replica set with the controller's own reconcile kernel, so a replica that already has the target shape is kept rather than bounced. Forcing a stuck-but-hydrating resize to commit therefore keeps the replica that was already up, and lands the record on `finalized` (forced) rather than `cancelled`, since the cut-over reached the record's own target. Creating a replica from the controller's `ReplicaShape` also drops a lossy round-trip through the planner's `ComputeReplicaConfig`, which cannot represent `INTROSPECTION DEBUGGING` without an interval. A cluster in that state used to get a replica whose logging disagreed with the config that called for it.
Follow-ups on deleting the staged reconfiguration machine, no behavior change. Collapse the cut-over routing's bool-plus-Option pair into a single optional cut-over target, computed unconditionally on that path. The fold with nothing in flight returns exactly the statement's shape, so applying it is the identity and the "nothing in flight, nothing to fold" special case disappears. The cut-over now computes its target through the same call the reshape path makes, so the two agreeing is structural rather than commented. Applying a target onto a managed config is the inverse of `realized_reconfiguration_target`, so it moves next to it in objects.rs as `apply_reconfiguration_target`, keeping the exhaustive-destructure compile guard beside the type it guards. `sequence_create_managed_cluster` builds the one `ReplicaShape` all its replicas share once, instead of a separate logging binding plus a per-iteration shape construction in the loop. Also fix the `sequence_alter_cluster_managed_to_managed` doc, which claimed the carried record is retired as cancelled and the replica set dropped wholesale: `retire_carried_reconfiguration` settles an on-target record as finalized (forced), and the reconcile keeps already-matching replicas.
Graceful cluster reconfiguration has been behind the `enable_zero_downtime_cluster_reconfiguration` feature flag, default off, so the `WITH (WAIT ...)` surface was rejected at plan time unless a deployment turned it on. That is now the only thing standing between an operator and the synchronous cut-over, which is the escape hatch for a reshape when the cluster controller itself is the problem. A break-glass path behind a default-off flag is not a break-glass path. Remove the flag and the planner gate. The two rejections that share that code path stay: a `WAIT` without a replica-shape change, and a `WAIT` on an unmanaged cluster. Every test that used the surface enabled the flag itself, so those statements go. In a mixed-version run some phases execute against a released binary that still enforces the gate (platform-checks' graceful-reconfiguration check runs its first manipulate phase there), so `get_minimal_system_parameters` pins the flag on below v26.38 instead. The docs drop the private-preview badges: the surface is generally available once nothing gates it.
aljoscha
marked this pull request as draft
August 7, 2026 11:49
aljoscha
force-pushed
the
aljoscha/cluster-legacy-06-compression-fix
branch
2 times, most recently
from
August 7, 2026 12:28
6ce5344 to
cf73bc0
Compare
Arrangement compression is a replica config shape dimension: it is part of `ReconfigurationTarget`, the reconcile kernel distinguishes replicas by it, and the controller transitions it through the hydrate-overlap like SIZE. Several hand-maintained dimension lists forgot it. The reshape path writes the reconfiguration record while resetting the config shape carried by `new_config` back to the realized values, deferring the transition to the record's cut-over. The reset enumerated size, replication factor, availability zones, and logging, but missed arrangement compression. A compression-carrying ALTER thus flipped the realized config immediately: the baseline strategy then desired the new shape and bounced the baseline replicas mid-reconfiguration, defeating the graceful overlap, and the cut-over later wrote the same value again. Reset via `realized_reconfiguration_target` and `apply_reconfiguration_target` instead of hand-listing dimensions, so a target dimension can no longer be missed here. The planner's WAIT gate had the same gap: it rejected a `WAIT` clause on a compression-only ALTER as having nothing to wait on, even though such an ALTER transitions through a hydrate-overlap like any other shape change. Allow it. The observability surface had it too: `mz_cluster_reconfigurations.changes` diffed only the other four dimensions, so a compression-only record reported an empty diff while in progress, and the SHOW CLUSTERS activity summary showed such a cluster as steady. Both now report the dimension. The compression ALTERs in managed_cluster.slt relied on the realized-config leak: their SHOW CREATE readback only saw the new value because the reshape wrote it early. They now request the synchronous cut-over (WAIT FOR '0s'), which advances the realized config in the ALTER's own transaction. ALTER ... RESET takes no WITH clause, so the RESET spelling is exercised for acceptance without a readback. cluster-controller.td gains regression coverage: the cc_preserve section asserts that a folding compression ALTER lands in the record's target and changes while the realized config stays untouched and only the overlap replica is bounced, and a new section runs a compression-only ALTER through the record path end to end. Changing the MV's definition changes its builtin fingerprint, so the builtin schema migration registers a replacement step for `mz_cluster_reconfigurations`, and the catalog-docs golden and the mz_show_clusters plan golden are rewritten.
aljoscha
force-pushed
the
aljoscha/cluster-legacy-06-compression-fix
branch
from
August 7, 2026 12:28
cf73bc0 to
8f11dd9
Compare
aljoscha
force-pushed
the
aljoscha/cluster-legacy-04-zdt-flag
branch
from
August 10, 2026 17:52
c6c719e to
d9f10e3
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #38104.
Arrangement compression is a replica config shape dimension: it is part of
ReconfigurationTarget, the reconcile kernel distinguishes replicas by it, and the controller transitions it through the hydrate-overlap likeSIZE. Several hand-maintained dimension lists forgot it.The reshape leak.
reshape_alter_cluster_managedwrites the reconfiguration record while resetting the config shape carried bynew_configback to the realized values, deferring the transition to the record's cut-over. The reset enumerated size, replication factor, availability zones, and logging, but missed arrangement compression. A compression-carryingALTERthus flipped the realized config immediately: the baseline strategy then desired the new shape and bounced the baseline replicas mid-reconfiguration, defeating the graceful overlap, and the cut-over later wrote the same value again. The reset now goes throughrealized_reconfiguration_target/apply_reconfiguration_targetinstead of hand-listing dimensions, so a target dimension can no longer be missed at this site.The planner gate. The
WAITgate rejected a compression-onlyALTERas having nothing to wait on, even though such anALTERtransitions through a hydrate-overlap like any other shape change. It is now allowed (and the error message names the option).The observability surface.
mz_cluster_reconfigurations.changesdiffed only the other four dimensions, so a compression-only record reported an empty diff while in progress, and theSHOW CLUSTERSactivity summary showed such a cluster as steady mid-reconfiguration. Both now report the dimension.Tests. The compression
ALTERs inmanaged_cluster.sltrelied on the leak: theirSHOW CREATEreadback only saw the new value because the reshape wrote it early. They now request the synchronous cut-over (WAIT FOR '0s'), which advances the realized config in theALTER's own transaction.ALTER ... RESETtakes noWITHclause, so theRESETspelling is exercised for acceptance without a readback.cluster-controller.tdgains regression coverage: thecc_preservesection asserts that a folding compressionALTERlands in the record's target andchangeswhile the realized config stays untouched and only the overlap replica is bounced, and a new section runs a compression-onlyALTERthrough the record path end to end.Known gap, left for a follow-up: the audit event's reconfiguration details intentionally omit compression, and a shape-dimension
RESETcannot carry aWAITclause (the parser only acceptsWITHonSET).Motivation
Fixes a bug in a feature that has yet to be released.
Checklist
$T ⇔ Proto$Tmapping (possibly in a backwards-incompatible way), then it is tagged with aT-protolabel.